Haptic functionality for network connected devices

ABSTRACT

A system provides haptic functionality over a networked system. The system receives information from a first device registered at the networked system and determines a notification to be provided to a user based on the information. The system then selects a second device registered at the networked system and provides the notification to the user by producing a haptic effect on the second device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/215,935, filed on Jul. 21, 2016, the specification of which is herebyincorporated by reference.

FIELD

One embodiment is directed generally to a haptic system, and inparticular, to a system that implements a network for providing hapticfunctionality.

BACKGROUND INFORMATION

“Haptics” relates to a tactile and force feedback technology that takesadvantage of the sense of touch of a user by applying haptic feedbackeffects (i.e., “haptic effects”), such as forces, vibrations, andmotions, to the user. Devices, such as mobile devices, touchscreendevices, and personal computers, can be configured to generate hapticeffects. In general, calls to embedded hardware capable of generatinghaptic effects (such as actuators) can be programmed within an operatingsystem (“OS”) of the device. These calls specify which haptic effect toplay. For example, when a user interacts with the device using, forexample, a button, touchscreen, lever, joystick, wheel, or some othercontrol, the OS of the device can send a play command through controlcircuitry to the embedded hardware. The embedded hardware then producesthe appropriate haptic effect.

SUMMARY

One embodiment is a system that provides haptic functionality over anetworked system. The system receives information from a first deviceregistered at the networked system and determines a notification to beprovided to a user based on the information. The system then selects asecond device registered at the networked system and provides thenotification to the user by producing a haptic effect on the seconddevice.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a computer server/system in accordance withan embodiment of the present invention.

FIG. 2 illustrates an example system for providing haptic functionalityaccording to an embodiment of the present invention.

FIG. 3 is a flow diagram of haptic functionality in accordance withembodiments of the present invention.

FIG. 4 illustrates another example system for providing hapticfunctionality according to an embodiment of the present invention.

FIG. 5 is a block diagram of an example cloud system for providinghaptic functionality in accordance with embodiments of the presentinvention.

FIG. 6 is a block diagram of another example system for providing hapticfunctionality in accordance with embodiments of the present invention.

FIG. 7 is a flow diagram of the operation of the haptic device decisionmodule of FIG. 6 in accordance with embodiments of the presentinvention.

FIG. 8 is a flow diagram of the operation of the haptic effect decisionmodule of FIG. 6 in accordance with embodiments of the presentinvention.

FIG. 9 is a flow diagram of the operation of the haptics module of FIG.1 when performing haptic functionality in accordance with embodiments ofthe present invention.

DETAILED DESCRIPTION

One embodiment provides haptic functionality for network connecteddevices such as objects in the Internet of Things (“IoT”). In oneembodiment, one or more haptically enabled devices in a networked systemare opportunistically selected and used to provide haptic effects to auser. In an alternative or additional embodiment, information collectedfrom one or more networked devices are used to provide haptic effects toa user. The haptic effects may be associated with a message, a service,or an alert that needs to be provided to the user. Accordingly, byselecting and using various network connected devices, embodiments allowfor better utilization of such devices in providing haptic functionalityto a user.

FIG. 1 illustrates a block diagram of a system 10 in accordance with oneembodiment of the invention. In one embodiment, system 10 is part of amobile device (e.g., a smartphone) or a non-mobile device, and system 10provides haptic functionality for the device. In another embodiment,system 10 is part of a device that is incorporated into an object incontact with a user in any way (e.g., furniture), and system 10 provideshaptic functionality for such a device. For example, in one embodiment,system 10 is part of a wearable device, and system 10 provides hapticfunctionality for the wearable device. Examples of wearable devicesinclude wrist bands, headbands, eyeglasses, rings, leg bands, arraysintegrated into clothing, or any other type of device that a user maywear on a body or can be held by a user. Some wearable devices can be“haptically enabled,” meaning they include mechanisms to generate hapticeffects. In another embodiment, system 10 is separate from the device(e.g., a mobile device or a wearable device), and remotely provideshaptic functionality for the device.

Although shown as a single system, the functionality of system 10 can beimplemented as a distributed system. System 10 includes a bus 12 orother communication mechanism for communicating information, and aprocessor 22 coupled to bus 12 for processing information. Processor 22may be any type of general or specific purpose processor. System 10further includes a memory 14 for storing information and instructions tobe executed by processor 22. Memory 14 can be comprised of anycombination of random access memory (“RAM”), read only memory (“ROM”),static storage such as a magnetic or optical disk, or any other type oftransitory or non-transitory computer-readable medium.

A computer-readable medium may be any available medium that can beaccessed by processor 22 and may include both a volatile and nonvolatilemedium, a removable and non-removable medium, a communication medium,and a storage medium. A communication medium may includecomputer-readable instructions, data structures, program modules, orother data in a modulated data signal such as a carrier wave or othertransport mechanism, and may include any other form of informationdelivery medium known in the art. A storage medium may include RAM,flash memory, ROM, erasable programmable read-only memory (“EPROM”),electrically erasable programmable read-only memory (“EEPROM”),registers, hard disks, removable disks, compact disk read-only memory(“CD-ROM”), or any other form of a storage medium known in the art.

In one embodiment, memory 14 stores software modules that providefunctionality when executed by processor 22. The modules include anoperating system 15 that provides operating system functionality forsystem 10, as well as the rest of a mobile device in one embodiment. Themodules further include a haptics module 16 that provides hapticfunctionality, as disclosed in more detail herein. In certainembodiments, haptics module 16 may comprise a plurality of modules,where each module provides specific individual functionality forproviding haptic effects. System 10 typically includes one or moreadditional application modules 18 to include additional functionality,such as TouchSense™ software by Immersion Corp.

System 10, in embodiments that transmit and/or receive data from remotesources, further includes a communication device 20, such as a networkinterface card, to provide mobile wireless network communication, suchas infrared, radio, Wi-Fi, cellular network communication, etc. In otherembodiments, communication device 20 provides a wired networkconnection, such as an Ethernet connection, a modem, etc.

Processor 22 is further coupled via bus 12 to a display 24, such as aLiquid Crystal Display (“LCD”), for displaying a graphicalrepresentation or user interface (“UI”) to a user. The display 24 may bea touch-sensitive input device, such as a touch screen, configured tosend and receive signals from processor 22, and may be a multi-touchtouch screen.

System 10, in one embodiment, further includes an actuator 26. Processor22 may transmit a haptic signal associated with a haptic effect toactuator 26, which in turn outputs haptic effects such as vibrotactilehaptic effects, electrostatic friction haptic effects, deformationhaptic effects, etc. Actuator 26 includes an actuator drive circuit.Actuator 26 may be, for example, an electric motor, an electro-magneticactuator, a voice coil, a shape memory alloy, an electro-active polymer,a solenoid, an eccentric rotating mass motor (“ERM”), a linear resonantactuator (“LRA”), a piezoelectric actuator, a high bandwidth actuator,an electroactive polymer (“EAP”) actuator, etc. In alternateembodiments, system 10 may include one or more additional actuators, inaddition to actuator 26 (not illustrated in FIG. 1). Alternatively oradditionally, actuator 26 may operate according to any other haptictechnology such as thermal displays (e.g., hot/cold), electrotactilestimulation (i.e., stimulation of tactile receptors with electriccurrent), kinesthetic feedback, etc. Yet another alternative oradditional embodiment may implement electrical muscle stimulations suchas a task that requires a user to determine what movement or movementsthe system is making them do and/or making them feel like doing.

Actuator 26 is an example of a haptic output device, where a hapticoutput device is a device configured to output any form of hapticeffects, such as vibrotactile haptic effects, electrostatic frictionhaptic effects, deformation haptic effects, etc., in response to a drivesignal. Accordingly, in alternate embodiments, actuator 26 may bereplaced by some other type of haptic output device (not shown) that maybe a non-mechanical or a non-vibratory device such as a device that useselectrostatic friction (“ESF”) or ultrasonic surface friction (“USF”), adevice that induces acoustic radiation pressure with an ultrasonichaptic transducer, a device that uses a haptic substrate and a flexibleor deformable surface or shape changing device and that may be attachedto a user's body, a device that provides projected haptic output such asa puff of air using an air jet, a laser-based projectile, a sound-basedprojectile, etc.

For example, one embodiment provides a laser-based projectile wherelaser energy ionizes air molecules in a concentrated region mid-air toprovide plasma (a concentrated mixture of positive and negativeparticles). In one embodiment, the laser may be a femtosecond laser thatemits pulses at very fast and very intense paces, and the faster thelaser, the safer for humans to touch. The projectile may appear as ahologram that is haptic and interactive. When the plasma comes incontact with user skin, the user may sense the vibrations of energizedair molecules in the concentrated region. Sensations on the user skinare caused by the waves that are generated when the user interacts withplasma in mid-air. Accordingly, haptic effects may be provided to theuser by subjecting the user to such concentrated region. Alternativelyor additionally, haptic effects may be provided to the user bysubjecting the user to the vibrations generated by directed soundenergy.

Further, in other alternate embodiments, system 10 may not includeactuator 26 or any other haptic output device, and a separate devicefrom system 10 includes an actuator or another haptic output device thatgenerates the haptic effects, and system 10 sends generated hapticsignals to that device through communication device 20.

In one embodiment, actuator 26 may be a “standard definition” (“SD”)actuator that generates vibratory haptic effects at a single frequency.Examples of an SD actuator include an ERM and an LRA. In contrast to anSD actuator, a high definition (“HD”) actuator or high fidelity actuatorsuch as a piezoelectric actuator or an EAP actuator is capable ofgenerating high bandwidth/definition haptic effects at multiplefrequencies. HD actuators are characterized by their ability to producewide bandwidth tactile effects with variable amplitude and with a fastresponse to transient drive signals. However, HD actuators have a largephysical dimension relative to SD actuators, and are more expensive thanSD actuators. Most devices consequently include only one or more SDactuators, instead of any HD actuators. Therefore, some embodiments mayleverage one or more speakers 28 in a devices in combination with the SDactuators to simulate HD haptic effects and provide an HD-like hapticexperience without the need for HD actuators.

System 10, in one embodiment, further includes a speaker 28. Processor22 may transmit an audio signal to speaker 28, which in turn outputsaudio effects. Speaker 28 may be, for example, a dynamic loudspeaker, anelectrodynamic loudspeaker, a piezoelectric loudspeaker, amagnetostrictive loudspeaker, an electrostatic loudspeaker, a ribbon andplanar magnetic loudspeaker, a bending wave loudspeaker, a flat panelloudspeaker, a heil air motion transducer, a plasma arc speaker, adigital loudspeaker, etc. In alternate embodiments, system 10 mayinclude one or more additional speakers, in addition to speaker 28 (notillustrated in FIG. 1). Further, in other alternate embodiments, system10 may not include speaker 28, and a separate device from system 10includes a speaker that outputs the audio effects, and system 10 sendsaudio signals to that device through communication device 20.

System 10, in one embodiment, further includes a sensor 30. Sensor 30may be configured to detect a form of energy, or other physicalproperty, such as, but not limited to, sound, movement, acceleration,biological signals, distance, flow, force/pressure/strain/bend,humidity, linear position, orientation/inclination, radio frequency,rotary position, rotary velocity, manipulation of a switch, temperature,vibration, visible light intensity, etc. Sensor 30 may further beconfigured to convert the detected energy, or other physical property,into an electrical signal, or any signal that represents virtual sensorinformation. Sensor 30 may be any device, such as, but not limited to,an accelerometer, a galvanic skin response sensor, a capacitive sensor,a hall effect sensor, an infrared sensor, an ultrasonic sensor, apressure sensor, a fiber optic sensor, a flexion sensor (or bendsensor), a force-sensitive resistor, a load cell, a LuSense CPS2 155, aminiature pressure transducer, a piezo sensor, a strain gauge, ahygrometer, a linear position touch sensor, a linear potentiometer (orslider), a linear variable differential transformer, a compass, aninclinometer, a magnetic tag (or a radio frequency identification(“RFID”) tag), a rotary encoder, a rotary potentiometer, a gyroscope, anon-off switch, a temperature sensor (such as a thermometer,thermocouple, resistance temperature detector, thermistor,temperature-transducing integrated circuit, etc.), a microphone, aphotometer, an altimeter, a biological monitor, a camera, alight-dependent resistor, etc., or any device that outputs anelectrocardiogram, an electroencephalogram, an electromyograph, anelectrooculogram, an electropalatograph, or any otherelectrophysiological output.

In alternate embodiments, system 10 may include one or more additionalsensors, in addition to sensor 30 (not illustrated in FIG. 1). In someof these embodiments, sensor 30 and the one or more additional sensorsmay be part of a sensor array, or some other type ofcollection/arrangement of sensors. Further, in other alternateembodiments, system 10 may not include sensor 30, and a separate devicefrom system 10 includes a sensor that detects a form of energy, or otherphysical property, and converts the detected energy, or other physicalproperty, into an electrical signal, or other type of signal thatrepresents virtual sensor information. The device may then send theconverted signal to system 10 through communication device 20.

Haptic Device Sharing Service

Generally, with known systems, a haptic device of a user is utilized toprovide haptic effects to that user. For example, wearable hapticdevices of a user (e.g., wrist bands, headbands, eyeglasses, rings, legbands, arrays integrated into clothing, etc.) can provide haptic effectscorresponding to state information intended for that user, such asincoming calls, navigational cues, messaging, etc.

The Internet of Things (“IoT”) is a network of objects (e.g., buildings,devices, vehicles, etc.) that include electronic functionality forcollecting and exchanging data. An increasing number of objects arebecoming connected to the IoT, including some objects that are touchedfrequently, such as chairs and tables in public places. At the sametime, haptic actuators are also making their way into many objects, suchas furniture (e.g., chairs, tables, etc.), architectural elements (e.g.,walls, floors, etc.), and various consumer objects such as rulers,staplers, etc. However, in known systems, providing haptic feedback to auser is generally limited to personal devices of that user or specialpurpose devices intended for a specific use. For example, in some knownsystems, haptic notifications are produced on devices that the userholds or wears such as a smart phone or a smart watch, or on specializeddevices configured for a specific use such as a D-Box seat in a movietheater.

In contrast to known systems, embodiments of the present inventionidentify available networked haptically enabled devices (e.g.,haptically enabled IoT devices) in the vicinity of a user and use themto provide haptic effects intended for that user. One embodimentprovides a service infrastructure that allows a haptically enableddevice to opportunistically produce haptic feedback for a user in itsenvironment. One embodiment further provides a brokerage system thatallows micro-transactions to be concluded between independent partiesfor using such haptically enabled devices, thus enabling a sharingeconomy for haptic feedback.

In one embodiment, as a user comes in close proximity to, or in contactwith, one or more IoT connected and haptically enabled objectsthroughout the day, each of these objects may be used to provide hapticfeedback to the user. For example, in a public place such as an airportor a coffee shop, objects that are touched frequently (e.g., chairs,tables, etc.) and are equipped with haptic actuators may be used tohaptically communicate with a nearby user, instead of or in addition tothe user's haptically enabled personal devices such as smart phones orwearables.

FIG. 2 illustrates an example system 200 for providing hapticfunctionality according to an embodiment of the present invention. Insystem 200, a user may be sitting on a haptically enabled chair 202 in apublic place. Chair 202 includes a haptic actuator 204 (e.g., an ERMactuator). Chair 202 also includes functionality to connect to the IoTor any other network. For example, chair 202 may include a WiFiconnection module for connecting to a WiFi network. Similarly, chair 202may include other modules for connecting to any other network describedherein with reference to various embodiments. Accordingly, actuator 204may be shared through the IoT and chair 202 may produce hapticnotifications to the user through actuator 204. Such hapticnotifications may be provided to the user instead of or in addition tohaptic notifications provided by a personal device of the user such as asmartphone in the user's pocket.

Alternatively or additionally, the user may be placing their hands orone of their personal devices (e.g., a laptop computer, a tablet, etc.)on a haptically enabled table 206 in a public place. Table 206 includesa haptic actuator 208 (e.g., an ERM actuator). Table 206 may alsoinclude functionality to connect to the IoT. Accordingly, actuator 208may be shared through the IoT and table 206 may produce hapticnotifications to the user through actuator 208.

One embodiment provides an infrastructure that enables communicationbetween the user's computing devices/services and the opportunistic,possibly publicly available, actuators in the user's environment. Suchinfrastructure may provide economic incentive to encourage setting upand sharing such opportunistic actuators, and this may in turn result inthese actuators to become more common in the environment.

An example of a haptically enabled device that may be opportunisticallyused is a stapler with a haptic actuator configured to vibrate when thestapler is empty. Any haptic actuators may be opportunistically used inhaptically enabled devices, such as actuators that are based on haptictechnologies that require contact (e.g., vibration, deformation,temperature, etc.) or those that are based on non-contact haptictechnologies (e.g., projected ultrasounds, air jets, static ESF, etc.).

One embodiment provides a network infrastructure that allows hapticactuators in a user's environment to be used opportunistically. Forexample, when a user sits on a chair, their smartphone may detect theavailability of a haptic actuator in the chair, establish communicationwith it, and use it to produce a haptic track for a movie that is playedback on the smartphone. In one embodiment, the haptified chair belongsto the user and is used exclusively by the user. In another embodiment,the haptified chair belongs to a third party, and the owner of the chairmay register it with a service such that its haptic actuator becomesavailable for others to use. The service can then managemicro-transactions between the haptic provider (i.e., the owner of thechair) and the user that desires haptic feedback. In one embodiment, theprovider may charge a fee (e.g., $0.001) for every notification producedon the haptic chair. The service thus enables a sharing economy forhaptic actuators, and therefore can encourage their spread in theenvironment.

In one embodiment, for example, a user may be sitting in a coffee shopand watching a movie on their tablet device. As the user presses “play,”a pop-up notification may appear asking the user if they would like touse the publicly available chair's haptic actuator for $1 per hour. Ifthe user presses “yes,” they may watch the movie while receivingcorresponding haptic effects from the chair.

In one embodiment, for example, a user's smartphone may determine thatthe user is late for a meeting. The smartphone may then identifyhaptically enabled objects that are likely to be touched by the user,and use them to provide haptic effects to notify the user. For example,a user may touch a haptically enabled stapler and feel a distinctvibration pattern that is provided based on communication with theuser's smartphone to indicate that the user is late for the meeting. Thesmartphone may produce such notifications on any haptically enabledobject that the user is likely to touch.

In one embodiment, a user is looking for an item in a supermarket andtypes a corresponding request on their smartphone to be guided towardthe correct aisle by haptic feedback on the floor. When the user walksin front of the correct aisle, the floor tiles vibrate to indicate tothe user to make a turn into that aisle. In an alternative or additionalembodiment, an air puff may be provided to guide the user into thecorrect aisle.

In one embodiment, a haptically enabled device that is dedicated to aperson, object, or task is opportunistically used to provide hapticeffects to a random person that is in its proximity. In one embodiment,the dedicated haptically enabled device implements a haptic technologythat facilitates such opportunistic functionality, such as projectionhaptics, floor haptics, etc. For example, an ultrasound emitter worn ona person's neck may be used to provide haptic notifications to anotherperson that is passing by. Similarly, a haptic floor pad intended for akiosk may be used to provide haptic feedback as a person walks on it. Inone embodiment, a distributed system detects when an opportunisticdevice is available in a person's environment, and a communicationplatform facilitates the sharing of the opportunistic device with thatperson.

In one embodiment, when a haptically enabled device that is dedicated toa first person is opportunistically used to provide haptic effects to asecond person that is in its proximity, a feedback (e.g., a haptic orother type of notification) is provided to the first person to indicatethat their device is being opportunistically used to provide hapticeffects to another person. For example, a smartphone of the first personmay provide textual, audio, visual, and/or haptic notification toindicate to the first person that one of their haptically enableddevices (e.g., an ultrasound emitter worn on the first person's neck) isbeing used to provide haptic effects to the second person. The firstperson may then have the option to ignore the notification ordisallow/terminate such opportunistic use.

In one embodiment, when an opportunistic haptically enabled device isavailable for providing haptic effects to a person, a wearable device ora mobile device of that person provides feedback to that person toindicate that haptic effects are being provided via the opportunisticdevice. In one embodiment, the feedback is provided with a distinctivehaptic effect. The feedback may also provide a directional cue toindicate where the opportunistic device is located. The feedback mayalso provide other haptic content associated with what is provided bythe opportunistic device. For example, the feedback may correspond to abackground haptic track that matches the haptic content provided by theopportunistic device.

One embodiment provides opportunistic haptic functionality byimplementing a haptic device in a user's environment, a user agent forrequesting haptic feedback from the haptic device, and a broker serviceto establish connections between the user agent and the haptic device.The user agent may be implemented anywhere in the system. For example,the user agent may be decoupled from the haptic device and may receiveinput from a variety of devices. In some embodiments, the user agent maybe implemented in the cloud, in a local network, etc. In somealternative or additional embodiments, a mobile device of a user mayinclude an actuator as well as a user agent for the mobile device and/orother devices such as devices in the user's device network (e.g.,wearables, haptically enabled personal items, etc.). Further details ofthe haptic device, the user agent, and the broker service are providedas follows.

Haptic Device

In one embodiment, one or more network connected haptic devices areembedded in a user's environment, for example, in furniture,architectural elements such as walls and floors, common objects,computational devices, etc. Each haptic device includes a hapticactuator, as well as functionality to communicate (e.g., wirelessly orthrough a wire) with user devices and/or web services. The haptic devicealso includes electronics and software in order to drive the hapticactuator.

The haptic actuator may provide haptic functionality according to anyembodiments described herein. For example, the haptic actuator mayprovide haptic feedback including vibration, deformation, heat, ESF, orany other type of haptic actuation. For example, a haptic device in auser's environment may be a table covered with a friction display suchthat varying textures can be felt when sliding a hand or finger on itssurface. As another example, a vibration actuator may be embedded intofloor tiles so that it can transmit vibrations to the feet of a user. Asanother example, an ultrasound emitter can be embedded in a wall so thatit can project haptic effects to nearby users. As another example, apeltier cell can be embedded in a door handle so that it can change itstemperature when touched by a user. Alternatively or additionally,smartphones, wearables, or other personal computing devices that includehaptic actuators can be used to provide haptic effects to the user.

The haptic device also includes functionality to communicate directly orindirectly with a user device or service regarding providingopportunistic haptic functionality. For example, the haptic device mayconnect to the Internet directly through a WiFi or other wirelessnetwork or indirectly through a gateway using Bluetooth or other IoTnetworking technologies. The haptic device may also connect directly tothe user's device using Bluetooth or other short range wirelesstechnologies. In some embodiments, the communication may first beestablished through near field communication (“NFC”), WiFi, or otherwireless technologies, and then switched to a more efficient short-rangetechnology such as Bluetooth.

In one embodiment, once the haptic device is connected to other devicesand services through wired or wireless communication, it can receivecommands for providing haptic effects. It may also communicate toprovide various information, for example, information about itscapabilities, its sensor readings, etc.

In one embodiment, the haptic device is discoverable by the user'sdevice and/or service so that the embodiment can determine when thehaptic device is within range of a user. For contact based hapticactuators (e.g., vibration actuators), the haptic device is within rangeof a user if the user is touching a surface that the actuator canproduce haptic effects on. For non-contact based haptic actuators (e.g.,ultrasound haptics), the haptic device is within range of a user whenthe user is within a certain distance of an emitter of the device andthere are no obstructions between the emitter and the user.

In one embodiment, instead of or in addition to determining which hapticdevices are within range of a user, it is determined which hapticdevices are in the vicinity of the user. This embodiment is applicable,for example, when it is difficult to determine with sufficient accuracywhether a haptic device is within range. For example, in one embodiment,without knowing which haptic devices are able to produce haptic effectsthat can be felt by the user, it is determined which haptic devices arenear the user's location. One embodiment may provide a list of nearbyhaptic devices to the user and let the user decide which device to usethrough a graphical UI.

In one embodiment, a user may explicitly request to use a specifichaptic device through an interaction with the haptic device or with theobject in which the haptic device is embedded (i.e., the haptifiedobject). In one embodiment, the user may interact with the haptifiedobject so that the interactions can be captured by sensors in the objectto select the object and/or to put the object in a discovery mode. Forexample, a user may tap their phone or wearable device against an NFCtag on a table or another haptified object to indicate that they desireto use haptic actuators of the table/object. For example, when the usertaps their phone against the object, the resulting vibrations and/orsounds can be captured by sensors on both the phone and the object andcompared to establish a connection between the haptified object and thephone (e.g., as in the contacts and file sharing “Bump” application fromBump Technologies). In one embodiment, the user may press a button onthe haptified object and a corresponding button on their phone toestablish a Bluetooth communication between the haptified object and thephone so that haptic actuators of the object may be used to providehaptic effects to the user.

In one embodiment, a projection haptic device may be paired with one ormore remote sensors such as cameras or depth sensors that can detect thepresence of a user within range of the haptic device. In one embodiment,the identity of the detected user may be determined by any knownfunctionality such as computer vision and a comparison with data storedin a database to establish a match. For example, one embodiment maycapture an image of the user using a camera, and use the image toperform facial recognition, read a name tag associated with the user,and/or otherwise identify the user. The embodiment may then communicatewith a server (e.g., a server for a social network, dating website,search engine, personal website, etc.) to determine additionalcharacteristics about the user. The characteristics may include, forexample, name, social security number, net worth, height, age, heritage,hair color, nationality, eye color, medical condition, credit score,gender, credit card number, username (e.g., for a website or account),password, temperament, mood, employer, job, hobby, likes, dislikes, etc.

In one embodiment, a haptic device that requires contact with a user toprovide haptic effects may use sensors to determine whether the user istouching a surface on which the device applies haptic effects. In oneembodiment, the surface may be equipped with a capacitive touch sensor,a pressure sensor, or other sensors that can detect a touch. In analternative or additional embodiment, an accelerometer may detectcontact with the body of the user and/or brushing against the surface.In one embodiment, for example, pressure sensors in a chair may be usedto detect whether a user is sitting on it, and this information may bepaired with other sensor data to determine the identity of the user thatis touching a surface when such touch has been detected.

In one embodiment, the haptified device includes an interaction sensorconfigured to detect an interaction with the device, and/or an objectthat includes the device, by a user (e.g., using a finger, foot, hand,arm, head, leg, or other body part). In one embodiment, the userinteraction may include touching the object, gesturing in real space,gesturing using the object (e.g., picking up the object and moving it inreal space), and/or gesturing on an object (e.g., swiping a finger alonga surface of the object). The interaction sensor is further configuredto generate a sensor signal associated with the interaction. Theinteraction sensor may include an accelerometer, a gyroscope, a camera,an RFID tag or reader, an indoor proximity system, a NFC communicationdevice, a global positioning system (“GPS”) device, a magnetometer, anultrasonic transducer, a wireless interface (e.g., an IEEE 802.11 orBluetooth interface), an infrared sensor, a depth sensor, and/or a rangesensor.

For example, in one embodiment, the interaction sensor includes awireless interface that is configured to detect the strength of awireless signal emitted by an object. The interaction sensor maygenerate a sensor signal associated with the wireless signal strength.Based on the wireless signal strength, the embodiment may determine, forexample, whether the sensor is within a predefined distance of theobject. If so, the embodiment may determine an interaction (e.g., comingwithin a predefined distance of the object) occurred. In one embodiment,the object may be carried by a person, and therefore proximity of theobject with the sensor would indicate proximity of the person with thesensor.

In another embodiment, the interaction sensor may be a 3D imaging systemthat is oriented toward the haptified device. Accordingly, theinteraction sensor may detect a user interaction (e.g., tap, touch,gesture on, shake, lift, gesture toward, etc.) with the device. Forexample, the interaction sensor includes a camera oriented toward thedevice. A user may make a gesture (e.g., a check mark sign) in the air(e.g., with a body part, such as a finger, hand, arm, foot, head, orleg) near the device. The interaction sensor may capture imagesassociated with the gesture and generate sensor signals. Based on thesensor signals, the embodiment may determine that a user interactionoccurred. The embodiment may further analyze the sensor signals todetermine the specific type of gesture that occurred. For example, theembodiment may analyze the sensor signals and determine that the usermade a check mark in the air with a finger.

In some embodiments, the interaction sensor is external to the haptifieddevice and in wired or wireless communication with the device. Forexample, the interaction sensor may include a camera associated with thedevice and in communication with the device. As another example, theinteraction sensor may comprise a 3D imaging system (e.g., the 3Dimaging system commonly sold under the trademark Microsoft Kinect® or anLED-based tracking system positioned external to the device (e.g., on ashelf in a store) and in communication with the device.

In some embodiments, the user may use an intermediary object (e.g., astylus, pen, cane, or wand) for an interaction with the haptifieddevice, and the embodiment may detect such an interaction.

One embodiment may detect a contact anywhere on the object that includesthe haptified device, or a contact with a specific location (e.g., alabel) on the object. The embodiment may detect a location of thecontact. For example, the embodiment may detect which portion of theobject (e.g., top, bottom, left side, right side, front, back, a label,an image, a logo, a piece of text, etc.) was contacted by the user. Inone embodiment, the user may gesture on a surface of the object tointeract with it. For example, the user may perform a two finger pinchon, move multiple fingers along, or make a checkmark on a surface of theobject.

In some embodiments, the user may interact with the object by making agesture using the object or a portion of the object. For example, theuser may move the object or a portion of the object in real 3D space(e.g., using the object or a portion of the object to draw a letter ornumber in the air, rotating, tilting, shaking, etc.). One embodiment maydetect more than one user interactions with the object (e.g., making agesture in front of the object, contacting the object, making agesturing along a surface of the object, and making a gesture using theobject). In some embodiments, a user interaction may include causing aninteraction between multiple objects. For example, the user may tap anobject against another object.

One embodiment may detect a user interacting with (e.g., tapping,holding, gesturing on, or gesturing toward) a first object with a firstbody part and interacting with a second object with a second body part.For example, the embodiment may detect the user interacting with thefirst object with the user's right hand and interacting with the secondobject with the user's left hand.

One embodiment may receive a first set of GPS data from the interactionsensor. The user may also carry a device that also includes GPSfunctionality and transmits a second set of GPS data. The embodiment maycompare the first set of GPS data with the second set of GPS data anddetermine the relative distance between the user and the haptifieddevice. If the user is within a predetermined distance from the device,the embodiment may determine that a user interaction has occurred.

The haptified device may further include one or more additional sensorsconfigured to generate sensor signals. In some embodiments, the sensorsmay include, for example, a humidity sensor, ambient light sensor,gyroscope, GPS unit, accelerometer, range sensor, depth sensor,biosensor, camera, or temperature sensor. In some embodiments, thesensors may be external to the device and in wired or wirelesscommunication with the device. For example, the sensors may include abiosensor coupled to a wearable device (e.g., a ring or wristband). Thebiosensor may be configured to wirelessly transmit sensor signals to thedevice, which may be, for example, positioned in the user's pocket.

In one embodiment, the interaction sensor includes a microphonepositioned to detect sounds associated with the manipulation of ahaptified object. For example, a user may shake a tin of coffee or a boxof cereal on a haptified table. The interaction sensor may detect soundsassociated with the shake. The interaction sensor may generate sensorsignals associated with the sounds. Based on the sensor signals, theembodiment may determine that a user interaction has occurred.

In one embodiment, the interaction sensor includes an optical sensor. Anobject may be positioned for blocking light from reaching theinteraction sensor. For example, the object may sit on top of aninteraction sensor embedded in a haptified table. Upon a user moving theobject or manipulating the object, the interaction sensor may detect achange in the amount of light. The interaction sensor may generatesensor signals associated with the change in the amount of light. Basedon the sensor signals, the embodiment may determine that a userinteraction occurred, or determine a characteristic of the userinteraction (e.g., if the user moved the object a distance that is abovea threshold).

One embodiment determines a characteristic (e.g., an amount of pressure,speed, direction, location, or type of gesture) associated with theinteraction. The embodiment analyzes sensor signals from the interactionsensor or any other sensors to determine the characteristic. Forexample, the embodiment may analyze images from a 3D imaging system todetermine a type of gesture (e.g., swipe, two-finger pinch, shake, etc.)made by the user.

One embodiment determines a characteristic associated with theinteraction using swept frequency capacitive sensing. Swept frequencycapacitive sensing may include measuring the change in the capacitanceof an object (e.g., in voltage) at a plurality of frequencies as a userinteracts with a haptified object. The embodiment may generate a profileof the user interaction based on the changed capacitances at theplurality of frequencies. One embodiment may compare the profile topredetermined interaction profiles. In one embodiment, eachpredetermined interaction profile may include a unique distribution ofchanges in capacitance at the plurality of frequencies and may beassociated with a different user interaction (e.g., a two finger pinch,a full hand grasp, or a single finger tap). For example, a predeterminedinteraction profile associated with a two finger pinch may be differentthan a predetermined interaction profile associated with a full handgrasp. Accordingly, the embodiment may determine specifically how theuser is interacting with the object.

One embodiment determines a characteristic associated with theinteraction based on sound signals from the interaction sensor. Forexample, the interaction sensor may include a microphone. The embodimentreceives sensor signals from the interaction sensor and compares datafrom the sensor signals with one or more sound profiles. In oneembodiment, each sound profile may include sound data that is associatedwith a different user interaction, for example, a user touching anobject with a finger, a knuckle, a finger nail, or a palm. For example,a sound profile associated with a finger contacting an object mayinclude a different frequency response than a sound profile associatedwith a knuckle contacting the object. Accordingly, the embodimentdetermines which body part of the user is interacting with the object.

One embodiment implements testing functionality to confirm whether ahaptified object is within range of a user. For example, after a userhas been determined to be in range of a haptified object, the object maybe tested to produce a haptic effect, and a user's device may be used todetect the haptic effect by a sensor such as an accelerometer.Accordingly, the embodiment may confirm that haptic effects generated bythe object can reach the user.

User Agent

One embodiment implements a user agent which is a device or service thatrepresents the user in requesting haptic feedback from haptic actuatorsin the user's environment. In one embodiment, the agent first determineswhen haptic feedback is required by the user and what haptic feedbackshould be produced. In one embodiment, for example, the user agentcollaborates with an application running on the user's phone todetermine when haptic feedback is required by the user and what hapticfeedback should be produced. For example, as the user watches a video ontheir device, a media player application may transmit the haptic trackof the video to the user agent. Similarly, the operating system of theuser's smartphone may transmit a haptic notification to the user agentfor playback.

The user agent then establishes a connection with a haptic actuator andrequests playback of a haptic effect or streaming of a sequence ofhaptic effects. The connection may be established according to any ofthe embodiments described herein. For example, the user agent may detectthe Bluetooth beacons of nearby haptic actuators and determine which oneis most likely to produce acceptable haptic feedback for the user.

In one embodiment, the communication between the user agent and thehaptic actuator may take place directly (e.g., using Bluetooth). In someembodiments, however, the communication may instead go through a thirdparty service that connects user agents with haptic actuators. In theseembodiments, the user agent may communicate with this service (e.g., aserver) to request a connection to a specific device or a list of nearbydevices. Communications can then continue indirectly through thisserver, or the user agent may later establish a direct connection to adevice through Bluetooth communication.

Broker Service

One embodiment uses a broker service to establish connections betweenuser agents and haptic actuators so that the haptic actuators may beshared by the users represented through the user agents. In oneembodiment, the service is a third party service that is providedindependently of haptic actuators and user agents. In one embodiment,the service is a web service that collects various information about thehaptic actuators such as their type, location, range, hapticcapabilities, etc. This information may be provided by the owner of thehaptic actuators in order to make them available through the service. Inone embodiment, the service tracks the location of mobile hapticactuators. For example, the service may track the location of the hapticactuators using GPS, indoor positioning data, cameras, etc. In oneembodiment, the service interacts indirectly with the haptic actuatorsusing a framework that allows for managing connected accessories/devicessuch as Apple's HomeKit, SmartThings hub, etc.

In one embodiment, when a user desires to find a shared device andreceive haptic feedback, a corresponding user agent communicates with abroker service to determine if any haptic actuators are available nearthe user and/or if the user is within range of any haptic actuators. Thebroker service may use the location of the user as well as any otheravailable sensor data (e.g., an interaction sensor or any other sensoraccording to any embodiments described herein) to determine whichactuators are near the user. The broker service may then provideavailability and other information (e.g., capability, type, etc.) of theavailable actuators to the user agent such that a connection can beestablished between the user agent and an identified actuator to let theuser receive haptic feedback from the haptic actuator.

In one embodiment, the broker service may include a payment system. Forexample, in one embodiment, a provider of an available haptic actuatormay indicate an associated cost (e.g., per unit of time, per hapticeffect, etc.) for using its haptic actuator. The cost may be fixed ormay change dynamically depending on various parameters such as time,competitor pricing, supply and demand, and other external factors in themarket. An example of a service that implements dynamic pricing is Uber.In one embodiment, the broker service collects the cost information fromthe providers of haptic actuators (e.g., price per haptic effect, perunit of time, etc.) so this information can be provided to a user agentthat inquires about shared haptic actuators. If the user agrees to paythe cost to receive haptic effects from a shared actuator, the brokerservice implements the payment system to receive payment from the userfor using the shared haptic actuator.

In one embodiment, the owner of a haptic actuator may determine a pricefor the use of the haptic actuator as a function of the time of day. Inthis embodiment, a user agent is informed of such price when making arequest to use the haptic actuator. In one embodiment, the user agentaccepts or rejects the service offer based on rules, such as the maximumprice set by the user represented by the user agent. In one embodiment,for example, the user agent may select the least expensive availablehaptic actuator or ask the user whether or not to accept a serviceoffer.

In one embodiment, the broker service and/or the user agent determinethe best shared haptic actuator to use based on the context of use. Forexample, in one embodiment, the haptic track of a movie may be betterprovided by HD actuators while a SD actuator may be sufficient for somehaptic notifications. In one embodiment, after identifying andcollecting information about available shared haptic actuators, thebroker service and/or the user agent may determine that the bestactuator to use is a haptic actuator of one of user devices of the user.

One embodiment identifies haptic actuators and makes them availablethrough a global network such as the IoT. In one embodiment, a brokerservice is implemented as a web service where haptic actuators registerin the cloud and are then available for other users that reach the cloudand desire to use available haptic actuators.

FIG. 3 is a flow diagram of haptic functionality in accordance withembodiments of the present invention. In one embodiment, thefunctionality of the flow diagram of FIG. 3 (and FIGS. 7-9 below) isimplemented by software stored in memory or other computer readable ortangible medium, and executed by a processor. In other embodiments, thefunctionality may be performed by hardware (e.g., through the use of anapplication specific integrated circuit (“ASIC”), a programmable gatearray (“PGA”), a field programmable gate array (“FPGA”), etc.), or anycombination of hardware and software.

At 302 a user device of a user indicates that a haptic notificationneeds to be provided to the user. For example, a smartphone of a usermay receive information or otherwise determine that the user needs to behaptically alerted of a notification.

At 304 the user device contacts a broker service with a request toprovide the haptic notification to the user.

At 306 the broker service identifies haptic actuators that are in theproximity of the user. For example, the broker service may use GPS orindoor positioning functionality of the user device and one or moreshared haptic actuators to determine if the user is nearby or withinrange of the haptic actuators.

At 308 the broker service offers a haptic actuator to the user device ata given price (e.g., per haptic notification, per unit of time, etc.).The price may have been set by a provider of the haptic actuators as afixed or dynamic price.

At 310 the user device accepts the offer to use the haptic actuator atthe given price.

At 312 the broker service establishes a communication between the userdevice and the haptic provider. In some embodiments, the haptic provideris implemented within the object that embeds the haptic actuator. Insome other embodiments, the haptic provider is separate from the hapticactuator, and the haptic actuator identifies/registers itself with thehaptic provider so that the haptic provider can determine what hapticeffects to send to the device/object that embeds the haptic actuator.

At 314 the haptic provider produces haptic feedback on the hapticactuator. Accordingly, the user receives the haptic notification via theshared haptic actuator of the haptic provider.

Haptic Alerts Based on Information from IoT Items or Mobile Devices

In one embodiment, various IoT items and/or mobile devices are utilizedto collect information pertinent to a user and to provide correspondinghaptic effects to the user. Examples of such items/devices aresmartphones, smart watches, smart wallets, smart cloths, smart fridges,or any other items that include functionality for connecting to a cloudsystem or to another network system and for providing information to thecloud system or the network system.

In one embodiment, the haptic effects may correspond to an alert that isnecessitated by the information collected from the IoT items and/ormobile devices. For example, information from the IoT items and/ormobile devices of a user may be processed to determine if the user ismissing or needing an item and provide corresponding alerts to the userif necessitated. The alert may be a haptic alert that is provided to theuser by a haptically enabled device/object of the user or by a sharedhaptic actuator as described herein with reference to variousembodiments.

FIG. 4 illustrates an example system 400 for providing hapticfunctionality according to an embodiment of the present invention.System 400 includes a user's suitcase 402 that can provide a hapticalert to the user based on information collected from the IoT itemsand/or mobile devices of the user. For example, the haptic alert may beprovided to the user by a haptic actuator 404 on a telescopic handle 406of suitcase 402 and/or by a haptic actuator 408 on a side handle 410 ofsuitcase 402. Actuators 404 and 408 may implement any hapticfunctionality described herein with reference to various embodiments.For example, actuators 404 and 408 may implement functionality accordingto TouchSense™ software by Immersion Corp.

In one embodiment, in addition to haptic actuator 408, side handle 410of suitcase 402 may include a weight sensor that can be used todetermine the weight of suitcase 402 when lifted by side handle 410 sothat a haptic notification may be provided to the user via hapticactuator 408 or via another haptically enabled device within range ofthe user to indicate if suitcase 402 is overweight.

In one embodiment, suitcase 402 may include various personal items ofthe user that are connected to the IoT, such as smart clothing or smartelectronic devices. The IoT items of the user may send information to acloud service which collects the information and compares it with a listof items that the user has indicated as required to being packed insuitcase 402. Such list may have been previously compiled and providedby the user to the cloud service. If the comparison indicates that theuser is missing an item that should have been included in suitcase 402,a distinctive haptic alert is provided to the user by haptic actuator404 on retractable handle 406 of suitcase 402 or by another hapticallyenabled device within range of the user.

In one embodiment, suitcase 402 may also include various sensors thatare used to collect information and provide haptic notification to theuser. For example, wheels 412 of suitcase 402 may include anacceleration sensor that can be used to determine if the user is on themove. Wheels 412 or any other portion of suitcase 402 may also includeGPS or other location functionality that can be used to determine thelocation of the user. Based on the information provided by theacceleration sensor, the location functionality, and/or the IoT items ormobile device, system 400 may provide any necessitated hapticnotifications to the user via haptic actuators 404 and/or 408 or viaanother haptically enabled device within range of the user. Oneembodiment may collect further information from IoT items located aroundthe user or synced manually with system 400.

One embodiment provides dynamic haptic alerts based on the location ofthe user. For example, if the user is lugging suitcase 402 at a shoppingarea of an airport and the boarding time is close, a haptic notificationmay be provided to the user via actuator 404 to indicate to the user toreturn to the gate. In another example, if the user is lugging suitcase402 at a shopping area of an airport and there is an item missing insuitcase 402, a haptic notification may be provided to the user viaactuator 404 to indicate to the user to purchase the missing item at theshopping are.

In one embodiment, after system 400 determines what items are packed insuitcase 402, it may further determine whether the user is moving (e.g.,out of a hotel). For example, system 400 may use an acceleration sensorand/or GPS functionality to determine movement that has an intensitythat cannot be attributed to just loading suitcase 402. Accordingly,when it is determined that suitcase 402 is moving, system 400 may alertthe user if suitcase 402 is missing an item. In one embodiment, theintensity of the alert is increased as the movement is increased. In oneembodiment, system 400 may provide an alert to the user via hapticactuator 404 or any other haptically enabled device within range of theuser.

One embodiment determines if the user is missing or needing an item byimplementing a “checksum” functionality over the items already with theuser. For example, a checksum functionality may be applied to the IoTitems in suitcase 402 to determine if the user is missing an item. Suchchecksum functionality may be used to dynamically create correspondinghaptic effects to be played back on a haptically enabled device that isclosest to the user and/or worn/held by the user, such as hapticactuator 404 of telescopic handle 406 of suitcase 402 or anotherhaptically enabled device within range of the user.

Embodiments are also applicable to any functionality for providingdynamic alerts based on the location of the user. For example, if a useris at a supermarket and their smart fridge indicates that they aremissing an item (e.g., milk), a user device or any other hapticallyenabled device within range of the user may be used to provide acorresponding haptic alert to the user. One embodiment furtherimplements functionality to further specify severity and/or priority ofsuch alerts.

One embodiment determines missing items by labeling items with keyfeatures. For example, one embodiment labels each item as a “bucket” ora “simple item,” where a “bucket” is an item that holds other items(e.g., a suitcase, a wallet, a purse, a backpack, etc.) and a “simpleitem” is an item that is placed inside a “bucket” (e.g., clothes,phones, computers, tablets, groceries, etc.). One embodiment furtherassociates relative priorities to the items. For example, a smart walletmay be given the highest priority while a certain piece of casual smartclothing may be given the lowest priority. The embodiment thendetermines, based on various inputs, (e.g., a user's schedule, location,etc.) which of their “items” they should have with them, and providesalerts based on missing items and the priority of missing items.

One embodiment implements a cloud based haptic control system, or otherremote network, that performs device authentication and registration inthe cloud, and user data storage of devices and their hapticcapabilities in the cloud. The cloud based haptic control system alsoexecutes device and effect decision functionality to determine whichdevice to play an effect on and which effect to play. The decisionalgorithms may be based on user inputs and usage context. Such userinputs may be provided by the same devices or by othercomponents/devices different than the devices.

In one embodiment, for example, the cloud system may be used toestablish a connection to a smart device to alert a user of the deviceif the device is left behind in a security line at an airport. Asanother example, the cloud system may be used to implement locationbased alerts (e.g., driving past a gas station, receiving an alert topurchase an item at a supermarket based on data from a smart fridge,etc.). As another example, the cloud system may be used to play a highpriority alert when a high priority item is stolen from a user's bag(e.g., a wallet/phone/watch is taken out of a bag).

In one embodiment, a cloud backend executes device and effect decisionfunctionality to control which device plays haptic effects and whathaptic effects to play. One embodiment provides device sidefunctionality for connectivity to the cloud and a software developmentkit (“SDK”) that allows the cloud to control haptic playback on adevice.

One embodiment implements a peer to peer network of devices where eachnode (e.g., each device) handles part of the execution of the device andeffect decision functionality instead of the cloud backend.

One embodiment tags alerts. By tagging different kinds of alerts, theembodiment may implement a lookup table of the tags and associatedalerts so that, upon receiving a tagged notification, the lookup tablecan be used to determine an alert haptic signal that corresponds to thetag and then relay the alert haptic signal to an appropriate hapticactuator.

FIG. 5 illustrates an example cloud system 500 for providing hapticfunctionality in accordance with embodiments of the present invention.In certain embodiments, services provided by cloud system 500 mayinclude a host of services that are made available to users of cloudsystem 500 on demand, such as online data storage and backup solutions,Web-based e-mail services, hosted office suites and documentcollaboration services, database processing, managed technical supportservices, and haptic services. Cloud computing is a model for enablingconvenient, on-demand network access to a shared pool of configurablecomputing resources (e.g., networks, servers, storage, applications, andservices). The services provided or accessed through the cloud (ornetwork) are referred to as cloud services. A cloud service provider maymake cloud services available to a customer using a device 502 or object504 that is registered at cloud system 500.

Services provided by cloud system 500 can dynamically scale to meet theneeds of its users. A specific instantiation of a service provided bycloud system 500 is referred to as a “service instance.” In general, anyservice made available to a user via a communication network, such asthe Internet, from a cloud service provider's system is referred to as a“cloud service.” Typically, in a public cloud environment, servers andsystems that make up the cloud service provider's system are differentfrom the customer's own personal and/or on-premises servers and systems.For example, a cloud service provider's system may host an application,and a user may, via a communication network such as the Internet, ondemand, order and use the application.

Cloud system 500 includes devices 502 and object 504 that implementfunctionality for connecting to a network 506. Devices 502 may includesmartphones, smart watches, smart fridges, etc. Objects 502 may includeany IoT objects described herein with reference to various embodiments.Network 506 may include one or more local area networks, wide areanetworks, the Internet, etc. Further, network 506 may include variouscombinations of wired and/or wireless networks, such as, for example,copper wire or coaxial cable networks, fiber optic networks, Bluetoothwireless networks, WiFi wireless networks, CDMA, FDMA, and TDMA cellularwireless networks, etc., which execute various network protocols, suchas, for example, wired and wireless Ethernet, Bluetooth, etc.

Devices 502 may include haptically enabled personal devices or any otherdevices that include haptic actuators. Objects 504 may also behaptically enabled and include one or more haptic actuators.

Cloud system 500 further includes a server 508 that is connected tonetwork 506. Server 508 may operate as a backend in cloud system 500 toprovide haptic functionality over cloud system 508. Cloud system 500 mayalso include a data store 510 that is used by server 508 to providebackend functionality over cloud system 500. Server 508 may execute aserver side application that communicates with client side applicationsexecuting on devices 502 and/or objects 504 to provide haptic effectsvia haptic actuators in devices 502 and/or objects 504. In oneembodiment, the server side application is a web server application thatcommunicates with a web browser application executing on a haptic device502. Other client-server and distributed software architectures are alsocontemplated by the present embodiments.

FIG. 6 illustrates an example system 600 for providing hapticfunctionality in accordance with embodiments of the present invention.System 600 implements device and effect decision functionality over acloud system such as cloud system 500 in FIG. 5. In system 600, one ormore user devices 602 provide various information (e.g., locationinformation, alerts, etc.) to a cloud backend service 604. Cloud backendservice 604 may be provided, for example, by server 508 and data store510 of cloud system 500 of FIG. 5.

Cloud backend service 604 includes a device registration andauthentication module 608 that allows for one or more haptic devices 606to connect to system 600, register with system 600, and beauthenticated. Haptic devices 606 may include haptically enabledpersonal devices (e.g., smartphones, smart watches, etc.) or hapticallyenabled IoT objects (e.g., chairs, tables, etc.). In one embodiment,haptic devices 606 may include haptically enables devices/objects thatare shared by their provider/user over system 600 so they can be used byother users.

Cloud backend service 604 also includes a user data module 610 thatreceives haptic device information from device registration andauthentication module 608 and determines whether a shared/availablehaptically enabled device/object is nearby or within range of a userthat desires to receive haptic effects. User data module 610 may use anyuser and/or device data available at cloud backend service 604 to makesuch determination. For example, user data module 610 may make suchdetermination based on the location of haptic devices 606 and thelocation of the user that desires to receive haptic effects or needs tobe notified with a haptic alert.

Cloud backend service 604 also includes a haptic device decision module612 that receives information about available haptic devices 606 fromuser data module 610 and determines which haptic devices 606 to use forproviding haptic effects to a user. Cloud backend service 604 alsoincludes a haptic effect decision module 614 that receives informationabout the selected haptic devices 606 from haptic device decision module612 and determines which haptic effects need to be provided on theselected haptic devices 606 and sends corresponding commands to theselected haptic devices 606.

FIG. 7 is a flow diagram of the operation of haptic device decisionmodule 612 of FIG. 6 in one embodiment. At 702 haptic device decisionmodule 612 receives information about available haptic devices 606 fromuser data module 610. At 704 haptic device decision module 612determines the proximity of haptic devices 606 to the user. At 706haptic device decision module 612 determines the haptic capabilities ofhaptic devices 606. Based on the proximity and haptic capabilities ofhaptic devices 606, at 708 haptic device decision module 612 determinesone or more subsets of haptic devices 606 to use for providing hapticeffects to the user.

For example, in one embodiment, haptic devices 606 may be divided intosubsets based on their properties (e.g., haptic capabilities, haptictechnology (e.g., vibration, projection, deformation, etc.), hapticeffect strength/perception, etc.). For example, one subset of hapticdevices 606 may include contact based haptic devices that are presumedto be in physical contact with the body of the user, and another subsetof haptic devices 606 may include non-contact based haptic devices thatare within range of the user. Alternatively or additionally, hapticdevices 606 may be divided into subsets based on their geographicallocation and/or the type/properties of the device/object they areembedded within (e.g., mobile/non-mobile devices, appliances,wearables/non-wearable devices, personal/public objects,secure/non-secure devices (e.g., whether the device is passwordprotected, etc.), battery-powered/line-powered devices, etc.).

FIG. 8 is a flow diagram of the operation of haptic effect decisionmodule 614 of FIG. 6 in one embodiment. At 802 haptic effect decisionmodule 614 receives information about one or more subsets of hapticdevices 606 determined by haptic device decision module 612 to use forproviding haptic effects to the user. At 804 haptic effect decisionmodule 614 determines a severity of a notification that needs to beprovided to the user. At 806 haptic effect decision module 614determines a fidelity of haptic devices in the subsets provided byhaptic device decision module 612. Based on the fidelity of the devicesand the severity of the notification, at 808 haptic effect decisionmodule 614 select a haptic device for providing the notification to theuser. Based on such a selection, at 810 haptic effect decision module614 selects a haptic effect for providing the notification to the uservia the selected haptic device.

In one embodiment, for example, the haptic effect may be selected from acloud library based on one or more input tags. For example, oneembodiment may implement an input tag corresponding to the device typeand another input tag corresponding to the alert type, and the inputtags in combination identify a haptic effect in the library that isconfigured to provide such an alert type (e.g., a strong vibration for ahigh priority alert) on such a device type (e.g., a haptic device thatincludes a low fidelity vibrating actuator). Once the haptic device andthe haptic effects are selected, corresponding information is providedto the haptic device to provide the haptic effect to the user.

FIG. 9 is a flow diagram of haptics module 16 of FIG. 1 when producinghaptic effects in accordance with embodiments of the present invention.

At 902 information is received from a first device registered at thenetworked system and at 904 a notification is determined to be providedto a user based on the information.

At 906 a second device registered at the networked system is selected.In one embodiment, the second device is selected from a plurality ofdevices registered at the networked system, and each one of theplurality of devices includes a haptic actuator. In one embodiment, thesecond device is selected based on a relative location of the seconddevice with respect to the user. In one embodiment, the relativelocation of the second device with respect to the user is determinedbased on the information received from the first device and informationat the networked system about the second device. In one embodiment, thesecond device is selected from the plurality of devices further based ona haptic capability of the second device, a severity of thenotification, and a fidelity of the second device. In one embodiment,the second device includes a sensor configured to detect a userinteraction with the second device, and the second device is selectedbased on the user interaction.

At 908 the notification is provided to the user by producing a hapticeffect on the second device. In one embodiment, the haptic effect isdetermined based on characteristics of the notification andcharacteristics of the second device. In one embodiment, thecharacteristics of the notification include a context of thenotification or a severity of the notification. In one embodiment, thecharacteristics of the second device include a fidelity of the seconddevice or a haptic capability of the second device.

In one embodiment, the information received from the first deviceindicates that the user is missing an item. In one embodiment, thenetworked system determines that the user is missing an item based oninformation received from a plurality of devices registered at thenetworked system. In one embodiment, the networked system determinesthat the user is missing an item and by comparing the informationreceived from the plurality of devices with a list of items that theuser needs to have, wherein the list is based on a location of the user.

As disclosed, embodiments allow for using various network connecteddevice such as IoT objects/devices to provide haptic effects to a user.In one embodiment, IoT devices are used to collect information pertinentto a user to alert the user if necessary. In one embodiment, networkconnected haptically enabled devices are opportunistically used to alerta user of a notification when the user is within range or in proximityof such devices. Accordingly, embodiments allow for better hapticnotification coverage.

Several embodiments are specifically illustrated and/or describedherein. However, it will be appreciated that modifications andvariations of the disclosed embodiments are covered by the aboveteachings and within the purview of the appended claims withoutdeparting from the spirit and intended scope of the invention.

What is claimed is:
 1. A non-transitory computer readable medium havinginstructions stored thereon that, when executed by a processor, causethe processor to provide haptic functionality over a networked systemby: receiving information from a first device registered at thenetworked system; determining a notification to be provided to a userbased on the information; detecting user interaction with a seconddevice registered at the networked system; selecting the second deviceto produce a haptic effect for the notification, wherein the seconddevice is selected based on the user interaction with the second device;and providing the notification to the user by causing the second deviceto produce the haptic effect.
 2. The non-transitory computer readablemedium of claim 1, wherein the second device includes a haptic actuatorconfigured to produce the haptic effect.
 3. The non-transitory computerreadable medium of claim 1, wherein the haptic effect is determinedbased at least in part on characteristics of the notification.
 4. Thenon-transitory computer readable medium of claim 3, wherein thecharacteristics of the notification include a context of thenotification or a severity of the notification.
 5. The non-transitorycomputer readable medium of claim 1, wherein the haptic effect isdetermined based at least in part on characteristics of the seconddevice.
 6. The non-transitory computer readable medium of claim 5,wherein the characteristics of the second device include a fidelity ofthe second device or a haptic capability of the second device.
 7. Thenon-transitory computer readable medium of claim 1, wherein the seconddevice includes a sensor configured to detect the user interaction withthe second device.
 8. The non-transitory computer readable medium ofclaim 1, wherein the information received from the first deviceindicates that the user is missing an item.
 9. The non-transitorycomputer readable medium of claim 1, wherein detecting the userinteraction comprises detecting the user touching a surface of thesecond device, such that the second device is selected in response todetecting the user touching the surface of the second device.
 10. Thenon-transitory computer readable medium of claim 1, wherein detectingthe user interaction with the second device is based on a near fieldcommunication (NFC) device included in the second device.
 11. A methodof providing haptic functionality over a networked system, the methodcomprising: receiving information from a first device registered at thenetworked system; determining a notification to be provided to a userbased on the information; detecting user interaction with a seconddevice registered at the networked system; selecting the second deviceto produce a haptic effect for the notification, wherein the seconddevice is selected based on the user interaction with the second device;and providing the notification to the user by causing the second deviceto produce the haptic effect.
 12. The method of claim 11, wherein thesecond device includes a haptic actuator configured to produce thehaptic effect.
 13. The method of claim 11, wherein the haptic effect isdetermined based at least in part on characteristics of thenotification.
 14. The method of claim 13, wherein the characteristics ofthe notification include a context of the notification or a severity ofthe notification.
 15. The method of claim 11, wherein the haptic effectis determined based at least in part on characteristics of the seconddevice.
 16. The method of claim 15, wherein the characteristics of thesecond device include a fidelity of the second device or a hapticcapability of the second device.
 17. The method of claim 11, wherein thesecond device includes a sensor configured to detect the userinteraction with the second device.
 18. The method of claim 11, whereinthe information received from the first device indicates that the useris missing an item.
 19. A system for providing haptic functionality overa networked system, comprising: a processor; and a memory coupled withthe processor and storing instructions that, when executed by theprocessor, cause the processor to: receive information from a firstdevice registered at the networked system; determine a notification tobe provided to a user based on the information; detect user interactionwith a second device registered at the networked system; select thesecond device to produce a haptic effect for the notification, whereinthe second device is selected based on the user interaction with thesecond device; and provide the notification to the user by causing thesecond device to produce the haptic effect.